Image processing apparatus, image processing method, and program

ABSTRACT

An image processing apparatus includes a first acquiring unit, a second acquiring unit, and a correction processor. The first acquiring unit acquires an intended viewing environment parameter, which is a parameter of an intended viewing environment, together with image data of a three-dimensional picture. The second acquiring unit acquires an actual viewing environment parameter, which is a parameter of an actual viewing environment for a user viewing the three-dimensional picture. The correction processor corrects the three-dimensional picture in accordance with a difference between the acquired intended viewing environment parameter and the acquired actual viewing environment parameter.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing apparatuses, imageprocessing methods, and programs, and more particularly, to an imageprocessing apparatus, an image processing method, and a program thatallow users to view three-dimensional (3D) pictures intended by acreator even if actual viewing environments are different from intendedviewing environments.

2. Description of the Related Art

Two-dimensional (2D) pictures are mainly used for video content,however, 3D pictures are attracting people's attention these days.Various display apparatuses and various encoding and decoding techniquesfor 3D pictures have been proposed (for example, see Japanese UnexaminedPatent Application Publication Nos. 10-327430 and 2008-182669).

A 3D picture includes a left-eye image viewed with a left eye and aright-eye image viewed with a right eye, and with a predetermined amountof parallax between a left-eye image and a right-eye image, viewers canperceive pictures three-dimensionally.

When creating 3D pictures, a content creator sets the amount of parallaxin advance by assuming user viewing environments, such as the viewingdistance, the binocular parallax, and the display size.

SUMMARY OF THE INVENTION

However, in many cases, actual viewing environments of a user aredifferent from intended viewing environments, which changes the depth of3D pictures. More specifically, the amount by which pictures pop out istoo large or too small. That is, pictures pop out or recede more thanthe creator intended. Additionally, the ratio between the depth and theplanar size on the screen may change, whereby a cubic shape looks like arectangular shape, thereby making 3D pictures distorted.

It is thus desirable to allow users to view 3D pictures as thoseintended by a creator even if actual viewing environments are differentfrom intended viewing environments.

According to an embodiment of the present invention, there is providedan image processing apparatus including first acquiring means, secondacquiring means, and correction processing means. The first acquiringmeans acquires an intended viewing environment parameter, which is aparameter of an intended viewing environment, together with image dataof a 3D picture. The second acquiring means acquires an actual viewingenvironment parameter, which is a parameter of an actual viewingenvironment for a user viewing the 3D picture. The correction processingmeans corrects the 3D picture in accordance with a difference betweenthe acquired intended viewing environment parameter and the acquiredactual viewing environment parameter.

According to another embodiment of the present invention, there isprovided an image processing method for an image processing apparatusincluding first acquiring means for acquiring data, second acquiringmeans for acquiring data, and correction processing means for correctinga 3D picture. The image processing method includes the steps of:acquiring by the first acquiring means an intended viewing environmentparameter, which is a parameter of an intended viewing environment,together with image data of the 3D picture; acquiring by the secondacquiring means an actual viewing environment parameter, which is aparameter of an actual viewing environment for a user viewing the 3Dpicture; and correcting by the correction processing means the 3Dpicture in accordance with a difference between the acquired intendedviewing environment parameter and the acquired actual viewingenvironment parameter.

According to still another embodiment of the present invention, there isprovided a program allowing a computer to execute processing includingthe steps of: acquiring an intended viewing environment parameter, whichis an intended parameter of a viewing environment, together with imagedata of a 3D picture; acquiring an actual viewing environment parameter,which is a parameter of an actual viewing environment for a user viewingthe 3D picture; and correcting the 3D picture in accordance with adifference between the acquired intended viewing environment parameterand the acquired actual viewing environment parameter.

According to an embodiment of the present invention, an intended viewingenvironment parameter is acquired together with image data of a 3Dpicture, and an actual viewing environment parameter for a user viewingthe 3D picture is also acquired. The 3D picture is corrected inaccordance with the difference between the acquired intended viewingenvironment parameter and the acquired actual viewing environmentparameter.

The program may be provided by being transmitted via a transmissionmedium or by being recorded on a recording medium.

The image processing apparatus may be an independent apparatus orelements forming an apparatus.

According to an embodiment of the present invention, it is possible toallow users to view 3D pictures intended by a creator even if an actualviewing environment parameter and an intended viewing environmentparameter are different.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of the configurationof a recording apparatus according to an embodiment of the presentinvention;

FIG. 2 is a flowchart illustrating a recording operation performed bythe recording apparatus shown in FIG. 1;

FIG. 3 illustrates a hierarchical structure of data recorded on arecording medium;

FIG. 4 illustrates an example of an extended partition in a movingpicture experts group phase 4 (MPEG4) box;

FIG. 5 illustrates another example of an extended partition in an MPEG4box;

FIG. 6 illustrates still another example of an extended partition in anMPEG4 box;

FIG. 7 is a block diagram illustrating an example of the configurationof a playback apparatus according to an embodiment of the presentinvention;

FIG. 8 is a flowchart illustrating a playback operation performed by theplayback apparatus shown in FIG. 7;

FIG. 9 is a block diagram illustrating an example of a correctionprocessor having a first configuration;

FIG. 10 illustrates generation of a virtual inter-camera distance image;

FIG. 11 illustrates a change in the amount of parallax when theinter-camera distance is changed;

FIG. 12 illustrates left-eye images and right-eye images before andafter being subjected to scaling processing;

FIG. 13 illustrates a change in the amount of parallax when the imagescaling amount is changed;

FIG. 14 illustrates variables necessary for determining the relationshipbetween the depth and each of the viewing environment parameters;

FIG. 15 illustrates the relationships of the changed depths Za, Zb, Zc,respectively, to the original depth Z when each of the viewingenvironment parameters is changed;

FIG. 16 illustrates the amount of correction when the viewing distanceis changed;

FIGS. 17A and 17B illustrate the amount of correction when the viewingdistance is changed;

FIG. 18 illustrates a summary of first correction processing;

FIG. 19 is a flowchart illustrating correction processing performed bythe correction processor having the first configuration;

FIG. 20 illustrates correction processing performed by the correctionprocessor having a second configuration;

FIG. 21 illustrates correction processing performed by the correctionprocessor having the second configuration;

FIGS. 22A, 22B, and 22C illustrate correction processing performed bythe correction processor having the second configuration;

FIG. 23 is a block diagram illustrating an example of the correctionprocessor having the second configuration;

FIG. 24 illustrates a summary of second correction processing;

FIG. 25 is a flowchart illustrating correction processing performed bythe correction processor having the second configuration; and

FIG. 26 is a block diagram illustrating an example of the configurationof a computer according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention are described below in thefollowing order:

-   1. Example of configuration of recording apparatus for producing    recording medium to be played back by a playback apparatus to which    an embodiment of the present invention is applied;-   2. Example of overall configuration of playback apparatus to which    an embodiment of the present invention is applied;-   3. First example of correction processor of playback apparatus;-   4. Second example of correction processor of playback apparatus; and-   5. Computer to which an embodiment of the present invention is    applied.    1. Example of Configuration of Recording Apparatus

FIG. 1 is a block diagram illustrating an example of the configurationof a recording apparatus 10 according to an embodiment of the presentinvention.

The recording apparatus 10 includes a video encoder 11, an audio encoder12, a multiplexer 13, and a recording controller 14.

The recording apparatus 10 encodes content data of 3D content andrecords the encoded content data on a recording medium 20, such as aBlu-ray (registered trademark) disc read only memory (BDROM). Thecontent data includes image data of 3D pictures (hereinafter referred toas “3D video data”) and audio data corresponding to the 3D video data.The 3D video data includes image data of a left-eye image and image dataof a right-eye image. The content data also includes, as additionalinformation (meta data), parameters of viewing environments intended bya content creator of 3D content (such parameters are hereinafterreferred to as the “intended viewing environment parameters”).

In embodiments of the present invention, viewing environment parametersinclude three types of parameters, i.e., the inter-eye distance of auser, the viewing distance between the user and a display unit, and thedisplay size of the display unit on which 3D pictures are displayed.

The video encoder 11 of the recording apparatus 10 encodes 3D video datainput from an external source according to an encoding method, such as amoving picture experts group phase 2 (MPEG2), an MPEG4, or an advancedvideo coding (AVC) method. The video encoder 11 supplies a video stream,which is an elementary stream (ES), obtained by encoding the 3D videodata to the multiplexer 13.

The audio encoder 12 encodes audio data corresponding to the 3D videodata input from an external source according to an encoding method, suchas an MPEG method, and supplies an audio stream, which is an ES,obtained by encoding the audio data to the multiplexer 13.

The multiplexer 13 combines the video stream supplied from the videoencoder 11 with the audio stream supplied from the audio encoder 12, andsupplies a multiplexed stream to the recording controller 14.

The recording controller 14 records the multiplexed stream supplied fromthe multiplexer 13 on the recording medium 20. The recording controller14 also records intended viewing environment parameters input from anoperation input unit (not shown) on the recording medium 20 as adefinition file.

In the recording apparatus 10 configured as described above, intendedviewing environment parameters, as well as 3D content data, are recordedon the recording medium 20 as additional information (meta data). Thismakes it possible for a playback apparatus to perform processing forcorrecting the content data for a difference between intended viewingenvironment parameters and actual viewing environment parameters whenplaying back the content data.

FIG. 2 is a flowchart illustrating a recording operation performed bythe recording apparatus 10 shown in FIG. 1. This recording operation isstarted, for example, when additional information, 3D video data, andaudio data are input.

In step S10, the recording controller 14 records intended viewingenvironment parameters, which serve as additional information, inputfrom an operation input unit on the recording medium 20 as a definitionfile. More specifically, three types of values, such as the inter-eyedistance, the viewing distance, and the display size, intended by acontent creator are recorded on the recording medium 20 as thedefinition file.

In step S11, the video encoder 11 encodes 3D video data input from anexternal source according to an MPEG method, such as MPEG2, MPEG4, orAVC. The video encoder 11 then supplies a video stream obtained byencoding the 3D video data to the multiplexer 13.

In step S12, the audio encoder 12 encodes audio data, which is inputfrom an external source, corresponding to the 3D video data according toan encoding method, such as MPEG. The audio encoder 12 then supplies anaudio stream obtained by encoding the audio data to the multiplexer 13.

In step S13, the multiplexer 13 combines the video stream supplied fromthe video encoder 11 with the audio stream supplied from the audioencoder 12. The multiplexer 13 then supplies the resulting multiplexedstream to the recording controller 14.

In step S14, the recording controller 14 records the multiplexed streamsupplied from the multiplexer 13 on the recording medium 20. Theoperation is then completed.

FIG. 3 illustrates a hierarchical structure of data to be recorded onthe recording medium 20.

The hierarchy of data recorded on the recording medium 20 includes, asshown in FIG. 3, layer A, layer B, and layer C. The layer C containsESs, such as an audio stream and a video stream. The layer B is a systemlayer having a multiplexed stream. The layer A contains information,which is the sole information recorded on the recording medium 20.

As described above, the intended viewing environment parameters, whichserve as additional information, are recorded on the layer A as the soledefinition file. However, the intended viewing environment parametersmay be recorded on the layer B or the layer C.

If additional information is recorded, for example, on the layer C, itis recorded as follows. If the encoding method is AVC, additionalinformation is recorded as supplemental enhancement information (SEI) oras part of a sequence parameter set (SPS) or a picture parameter set(PPS). If the encoding method is MPEG2, additional information isrecorded as a video sequence or extension and user data(extension_and_user_data).

In this case, additional information may be variable within one videostream. Additionally, even if a plurality of 3D picture video streamshaving different intended viewing environment parameters are recorded onthe recording medium 20, each intended viewing environment parameter canbe supplied to a playback apparatus.

If additional information is recorded on the layer B, it is recorded on,for example, a private packet of a transport stream (TS), a private packof a program stream (PS), or an extended partition of a box contained inMPEG4 configuration information.

The extended partition of an MEPG4 box on which additional informationis recorded is provided in a private extension box (shown as “uuid” inFIG. 4) immediately after a ftyp box, which is positioned at the head ofa file. In this case, a playback apparatus that plays back 3D video datarecorded on the recording medium 20 can obtain intended viewingenvironment parameters before performing decoding. However, the intendedviewing environment parameters are invariable within the file.

In the private extension box, in addition to intended viewingenvironment parameters, the type of codec, the bit rate, the frame size,the aspect ratio, information concerning whether an image is a 2Dpicture or a 3D picture, and so on, are recorded.

The extended partition of the MPEG4 box may be provided, as shown inFIG. 5, in a partition (shown as “stsd” in FIG. 5) of track information(shown as “trak” in FIG. 5) within a moov box. In this case, informationindicating in which partition additional information is recorded iscontained in a video stream. Based on this information, a playbackapparatus obtains intended viewing environment parameters. In this case,therefore, the intended viewing environment parameters are variablewithin the video stream. However, the accessibility is poorer than thatof a case where intended viewing environment parameters are recorded asshown in FIG. 4.

Alternatively, the extended partition of the MPEG4 box may be providedin an mdat box, as shown in FIG. 6. That is, additional information maybe recorded as one media stream (side info.stream). In this case, avideo stream and additional information are synchronized with each otherin accordance with time information, and thus, intended viewingenvironment parameters can be changed every moment.

In the examples shown in FIGS. 4 through 6, after the ftyp box, the moovbox and the mdat box are disposed in this order. However, thearrangement of the moov box and the mdat box is not restricted to this.

2. Example of Overall Configuration Playback Apparatus

FIG. 7 is a block diagram illustrating an example of the configurationof a playback apparatus 50 according to an embodiment of the presentinvention.

The playback apparatus 50 shown in FIG. 7 includes a reader 51, ademultiplexer 52, a video decoder 53, a correction processor 54, anoperation input unit 55, and an audio decoder 56. The playback apparatus50 plays back, together with additional information, 3D video data andcorresponding audio data recorded on the recording medium 20, andsuitably displays 3D pictures on the basis of the additionalinformation.

More specifically, the reader 51 of the playback apparatus 50 readsintended viewing environment parameters recorded on the recording medium20 as additional information, and supplies the intended viewingenvironment parameters to the correction processor 54. The reader 51also reads a multiplexed stream recorded on the recording medium 20, andsupplies the multiplexed stream to the demultiplexer 52.

The demultiplexer 52 separates the multiplexed stream supplied from thereader 51 into a video stream and an audio stream. The demultiplexer 52supplies the video stream to the video decoder 53 and also supplies theaudio stream to the audio decoder 56.

The video decoder 53 decodes the video stream supplied from thedemultiplexer 52 according to a method corresponding to the encodingmethod used in the video encoder 11 shown in FIG. 1, and supplies theresulting 3D video data to the correction processor 54.

The correction processor 54 performs 3D picture correction processingfor correcting the 3D video data supplied from the video decoder 53 inaccordance with the difference between the intended viewing environmentparameters supplied from the reader 51 and actual viewing environmentparameters supplied from the operation input unit 55. Then, thecorrection processor 54 outputs the corrected 3D video data, i.e.,left-eye image data and right-eye image data, to a display unit 61.

The operation input unit 55 receives an input of actual viewingenvironment parameters from a user who views 3D pictures. The userinputs the actual viewing environments by using the operation input unit55. More specifically, three types of values, such as the inter-eyedistance, the viewing distance, and the display size, similar to thoseof the intended viewing environment parameters, are input by the user.The operation input unit 55 supplies the actual viewing environmentparameters input by the user to the correction processor 54.

The audio decoder 56 decodes the audio stream supplied from thedemultiplexer 52 according to a method corresponding to the encodingmethod used in the audio encoder 12 shown in FIG. 1, and supplies theresulting audio data to a speaker 62.

The display unit 61 displays a left-eye image and a right-eye imagecorresponding to the 3D video data supplied from the correctionprocessor 54, for example, in a time-division multiplexing manner. Inthis case, a viewer wears, for example, shutter glasses that synchronizewith the switching of the left-eye image and the right-eye image, andobserves the left-eye image only with the left eye and observes theright-eye image only with the right eye. This makes it possible for theuser to perceive 3D pictures three-dimensionally.

The speaker 62 outputs sound corresponding to the audio data suppliedfrom the audio decoder 56.

FIG. 8 is a flowchart illustrating a playback operation performed by theplayback apparatus 50. This playback operation is started, for example,when an instruction to play back 3D content recorded on the recordingmedium 20 is given by a user.

In step S31, the operation input unit 55 receives an input of actualviewing environment parameters from a user who views 3D pictures, andsupplies the received actual viewing environment parameters to thecorrection processor 54. Step S31 may be executed in advance separatelyfrom the following steps. That is, actual viewing environment parametersmay be input in advance, for example, on a setting screen, before aninstruction to play back 3D content is given.

In step S32, the reader 51 reads intended viewing environment parametersrecorded on the recording medium 20 as additional information, andsupplies the intended viewing environment parameters to the correctionprocessor 54.

In step S33, the playback apparatus 50 reads a multiplexed stream of 3Dcontent recorded on the recording medium 20 and decodes the multiplexedstream. That is, the reader 51 reads the multiplexed stream of the 3Dcontent from the recording medium 20, and supplies the multiplexedstream to the demultiplexer 52. The demultiplexer 52 separates themultiplexed stream into a video stream and an audio stream. The videodecoder 53 decodes the video stream according to a method correspondingto the encoding method used in the recording apparatus 10 and suppliesthe resulting 3D video data to the correction processor 54. The audiodecoder 56 decodes the audio stream according to a method correspondingto the encoding method used in the recording apparatus 10 and suppliesthe resulting audio data to the speaker 62.

In step S34, the correction processor 54 performs 3D picture correctionprocessing for correcting the supplied 3D video data in accordance withthe difference between the intended viewing environment parameterssupplied from the reader 51 and the actual viewing environmentparameters supplied from the operation input unit 55. The correctionprocessing includes first correction processing and second correctionprocessing. Details of the first and second correction processing arediscussed later.

In step S35, the correction processor 54 outputs the corrected 3D videodata, i.e., the image data of the left-eye image and the right-eyeimage, to the display unit 61. Also in step S35, the audio decoder 56outputs the audio data corresponding to the corrected 3D video data tothe speaker 62. When the 3D content pictures and sound to be played backare entirely output, the playback operation is completed.

As described above, the playback apparatus 50 corrects for thedifference between viewing environments intended by a content creatorand actual viewing environments input by a user, thereby making itpossible to provide 3D pictures without distortions, which wouldotherwise be caused by the difference of viewing environments.

Details of 3D picture correction processing performed by the correctionprocessor 54 are given below. In the playback apparatus 50, a firstconfiguration of the correction processor 54 shown in FIG. 9 or a secondconfiguration of the correction processor 54 shown in FIG. 23 may betaken.

3. First Example of Correction Processor of Playback Apparatus

FIG. 9 is a block diagram illustrating an example of a firstconfiguration of the correction processor 54.

The correction processor 54 shown in FIG. 9 includes a parallax controlparameter calculator 81 and a parallax controller 82. The parallaxcontroller 82 includes a parallax detector 91, a virtual inter-cameradistance image generator 92, and an image scaling unit 93.

Intended viewing environment parameters input from the reader 51 andactual viewing environment parameters input from the operation inputunit 55 are supplied to the parallax control parameter calculator 81.The parallax control parameter calculator 81 calculates two parallaxcontrol parameters, such as a virtual inter-camera distance and anamount by which images are scaled (such an amount is hereinafterreferred to as the “image scaling amount”), in accordance with thedifference between the intended viewing environment parameters and theactual viewing environment parameters.

Among the viewing environment parameters, if the viewing distance isdifferent between the intended viewing environment parameters and theactual viewing environment parameters, the virtual inter-camera distancecan be controlled (changed) so that distortions of a 3D picture arecorrected. This is described in detail below.

If the inter-eye distance or the display size is different between theintended viewing environment parameters and the actual viewingenvironment parameters, one of or both the virtual inter-camera distanceand the image scaling amount can be controlled (changed) so thatdistortions of a 3D picture are corrected. If both the virtualinter-camera distance and the image scaling amount are used, the controlratio of the virtual inter-camera distance to the image scaling amountis determined. Then, in accordance with the determined control ratio,parallax control parameters corresponding to the amount of correctionmade by using the virtual inter-camera distance and the amount ofcorrection made by using the image scaling amount are calculated. Thecontrol ratio of the virtual inter-camera distance to the image scalingamount may be stored in the correction processor 54 as a fixed value, ormay be input from, for example, the operation input unit 55.

The parallax detector 91 detects the amount of parallax between theleft-eye image and the right-eye image of the 3D video data suppliedfrom the video decoder 53 (FIG. 7), and more specifically, the parallaxdecoder 91 detects the amount of parallax between each pixel of theleft-eye image and the corresponding pixel of the right-eye image byusing, for example, a block matching method. The parallax detector 91creates a parallax map by setting the amounts of parallax ofcorresponding pixels between the left-eye image and the right-eye imageas luminance levels, and supplies the parallax map to the virtualinter-camera distance image generator 92. It is not necessary that theamounts of parallax be represented by a parallax map, and the amounts ofparallax may be shown in any form as long as the virtual inter-cameradistance image generator 92 can identify the detected amounts ofparallax.

A left-eye image and a right-eye image, which form an original image(before being corrected) captured with a predetermined inter-cameradistance, are supplied to the virtual inter-camera distance imagegenerator 92 from the video decoder 53 (FIG. 7). The amount of parallaxbetween each pixel of the original left-eye image and the correspondingpixel of the original right-eye image is supplied to the virtualinter-camera distance image generator 92 from the parallax detector 91as a parallax map. The virtual inter-camera distance image generator 92generates a left-eye image and a right-eye image that would be capturedwith a virtual inter-camera distance. That is, the virtual inter-cameradistance image generator 92 generates an image that would be capturedwith the virtual inter-camera distance (such an image is hereinafterreferred to as the “virtual inter-camera distance image”) supplied fromthe parallax control parameter calculator 81 by using the originalleft-eye image, the original right-eye image, and the amounts ofparallax between the left-eye image and the right-eye image.

The virtual inter-camera distance supplied from the parallax controlparameter calculator 81 is expressed by the ratio of the virtualinter-camera distance to the original inter-camera distance. Forexample, as shown in FIG. 10, assuming that the inter-camera distanceused when the original left-eye image and the right-eye image arecaptured is 1, the camera position for the original left-eye image isset to be 0.0, and the camera position for the original right-eye imageis set to be 1.0.

When a virtual inter-camera distance of 0.5 is supplied from theparallax control parameter calculator 81, the virtual inter-cameradistance image generator 92 generates, as shown in FIG. 10, a left-eyeimage having a virtual camera position of 0.25 and a right-eye imagehaving a virtual camera position of 0.75. In this case, the ratio of theinter-camera distance for the generated left-eye image and right-eyeimage to the original inter-camera distance is 0.5.

FIG. 11 illustrates a change in the amount of parallax when theinter-camera distance is changed. The positive sides of the coordinatesindicate the pop-out direction (forward direction, i.e., toward theviewer) of pictures, while the negative sides of the coordinatesindicate the receding direction (backward direction, i.e., away from theviewer) of pictures.

When the inter-camera distance is reduced to 50% (0.5) of the originalinter-camera distance, the amount of parallax is also reduced to ½ theamount of parallax before being corrected. When the inter-cameradistance is increased to 200% (2.0) of the original inter-cameradistance, the amount of parallax is also doubled.

Referring back to FIG. 9, the image scaling unit 93 scales a left-eyeimage and a right-eye image supplied from the virtual inter-cameradistance image generator 92 in accordance with the image scaling amountsupplied from the parallax control parameter calculator 81. The imagescaling amount supplied from the parallax control parameter calculator81 is expressed by the ratio of the image scaling amount to the originalimage scaling amount. The image scaling unit 93 then outputs the imagedata of the left-eye image and the right-eye image after being scaled tothe display unit 61 (FIG. 7).

FIG. 12 illustrates left-eye images and right-eye images before andafter being subjected to scaling processing.

In FIG. 12, the left-eye image and the right-eye image before beingscaled, illustrated as the original images, are corrected with a scalingamount (scaling ratio) of 0.5 in the direction in which parallax occurs,i.e., in the horizontal direction. The resulting left-eye image andright-eye image are shown as the scaled images.

FIG. 13 illustrates a change in the amount of parallax when the scalingamount is changed.

When the scaling amount is reduced to 50% (0.5) of the original scalingamount, the amount of parallax is also reduced to ½ the amount ofparallax before being corrected. When the scaling amount is increased to200% (2.0) of the original scaling amount, the amount of parallax isalso doubled.

A description is given below of the relationship between a change in thedepth and a change in each of the viewing environment parameters, suchas the viewing distance, the inter-eye distance, and the display size.

Variables necessary for determining the relationship between the depthand each of the viewing environment parameters are defined withreference to FIG. 14.

It is now assumed that the inter-eye distance of a user viewing 3Dpictures is designated by E and that the distance from the user to adisplay unit which displays 3D pictures (such a distance is referred toas the “viewing distance”) is indicated by L. It is now assumed that, inthe 3D picture viewed by the user, a pixel x_(L) at one position of theleft-eye image corresponds to a pixel x_(R) at the associated positionof the right-eye image. That is, the user views a 3D picture with anamount of parallax d=(x_(L)−x_(R)). In this case, the depth reproducedon the user's retina is represented by Z.

In this case, considering the relationship between the depth Z and theamount of parallax d, Z can be expressed by equation (1):E/(L−Z)=d/ZZ=L·d/(E+d)   (1)where, throughout the specification and the drawings, “·” in equationsmeans multiplication.

The relationship between the viewing distance L and the depth Z is asfollows.

The depth Za when the viewing distance is changed to the viewingdistance L′, which is changed from the original viewing distance L by afactor S, i.e., L′=S·L, is now considered.

From equation (1), Za=L′·d/(E+d) holds true. Since L′=S·L, equation (2)is established.Za=S·L·d/(E+d)   (2)Equation (2) can be expressed by equation (3).Za=S·Z   (3)Accordingly, when the viewing distance is changed by a factor S, thedepth (pop-out or receding amount) Za is also changed from the originaldepth Z by a factor S.

The relationship between the display size and the depth Z is as follows.

When the display size is changed from the original display size by afactor S, the amount of parallax d′ is also changed by a factor S(d′=S·d). From equation (1), the depth Zb when the amount of parallaxd′=S·d can be expressed by Zb=L·d′/(E+d′). Since d′=S·d, equation (4)holds true.Zb=L·S·d/(E+S·d)   (4)

By eliminating d in equation (4) by using equation (1), the followingequation (5) is established.Zb=L·S·Z/(L−Z+S·Z)   (5)

Accordingly, when the display size is changed from the original displaysize by a factor S, the depth Zb is nonlinear with respect to theoriginal depth Z, as expressed by equation (5).

The relationship between the depth Z and the inter-eye distance is asfollows.

The depth Zc when the inter-eye distance is changed from the originalinter-eye distance E by a factor S (E′=S·E) is now considered. Fromequation (1), Zc=L·d/(E′+d) holds true. Since E′=S·E, equation (6) canbe established.Zc=L·d/(S·E+d)   (6)

By eliminating d in equation (6) by using equation (1), the followingequation (7) holds true.Zc=L·Z/(S·L−S·Z+Z)   (7)

Accordingly, when the inter-eye distance is changed from the originalinter-eye distance E by a factor S, the depth Zc is nonlinear withrespect to the original depth Z, as expressed by equation (7).

FIG. 15 illustrates examples of equations (3), (5), and (7) expressingthe relationships of the changed depths Za, Zb, Zc, respectively, withrespect to the original depth Z when the viewing distance, the displaysize, and the inter-eye distance are changed.

In FIG. 15, the original viewing environment parameters are as follows:the viewing distance L is 1500 mm; the display size is 42 inches; andthe inter-eye distance E is 65 mm.

Equation (3) in FIG. 15 represents the relationship of the changed depthZa with respect to the original depth Z when the viewing distance L isincreased to 2250 mm.

Equation (5) in FIG. 15 represents the relationship of the changed depthZb with respect to the original depth Z when the display size isincreased to 63 inches.

Equation (7) in FIG. 15 represents the relationship of the changed depthZc with respect to the original depth Z when the inter-eye distance E isincreased to 97.5 mm.

A description is now given of the amounts of correction, i.e., thevirtual inter-camera distance and the image scaling amount, inconsideration of the above-described relationship between change indepth and change in viewing environment parameters.

The amount of correction when the viewing distance is changed by afactor S, i.e., L′=S·L is first described.

Assuming that the depth Za when the viewing distance L is changed to theviewing distance L′ is indicated by Z′, equation (2) can be expressed byequation (8).Z′(d)=S·L·d/(E+d)   (8)

The equation for correcting the amount of parallax d that makes thechanged depth Z′ be the same as the original depth Z is considered.Assuming that the corrected amount of parallax is designated by d′,equation (9) is defined as follows.d′=f(d)   (9)

By changing the amount of parallax d in equation (8), equation (10) isobtained.Z′(d′)=S·L·d′/(E+d′)   (10)

Considering the equation d′=f(d) that makes the changed depth Z′ be thesame as the original depth Z, since Z=Z′, equations (11) areestablished.Z=Z′L′·d/(E+d)=S·L·d′/(E+d′)S·d′·(E+d)=d·(E+d′)d′·(S·E+S·d−d)=d·Ed′=d·E/(S·E+S·d−d)   (11)

Accordingly, when the viewing distance is changed from the originalviewing distance by a factor S, to cancel such a change in the viewingdistance, the amount of parallax is changed to the amount of parallax d′obtained by nonlinear equations (11).

FIG. 16 illustrates equations (11) in a conceptual drawing when theviewing distance is changed (L′/L) to 200% and 50% of the originalviewing distance.

In this embodiment, the virtual inter-camera distance is given as aparallax control parameter. Accordingly, when the viewing distance ischanged from the original viewing distance by a factor S, in order tocancel such a change, the virtual inter-camera distance C(d) expressedby equation (12) is given as the amount of correction.C(d)=d′/d   (12)

FIG. 17A illustrates equations (11) when the viewing distance L isdoubled under the viewing environments in which the viewing distance Lis 1500 mm, the display size is 42 inches, and the inter-eye distance Eis 65 mm.

FIG. 17B illustrates equations (11) shown in FIG. 17A in the form of thevirtual inter-camera distance expressed by equation (12) C(d)=d′/d.

The amount of correction when the display size is changed from theoriginal display size by a factor S is discussed below.

When the display size is changed by a factor S, the amount of parallaxis also changed by a factor S. Accordingly, the virtual inter-cameradistance or the image scaling amount that changes the amount of parallaxto 1/S the original amount of parallax is set as the amount ofcorrection. For example, if the amount of parallax is increased to fourtimes, the virtual inter-camera distance is reduced to 25% of theoriginal virtual inter-camera distance or the image scaling amount isreduced to 25% of the original image scaling amount.

Alternatively, a combination of the virtual inter-camera distance andthe image scaling amount may be set as the amount of correction. Forexample, if the control ratio of the virtual inter-camera distance tothe image scaling amount is 50:50, a 3D image with a virtualinter-camera distance of 50% may be generated, and then, image scalingprocessing may be performed on the generated 3D image with an imagescaling ratio of 50%.

The amount of correction when the inter-eye distance is changed by afactor S (E′=S·E) is discussed below.

As in the case of the viewing distance described above, assuming thatthe depth Zc when the inter-eye distance is changed to the inter-eyedistance E′ is indicated by Z′, equation (6) can be expressed byequation (13).Z′(d)=L·d/(S·E+d)   (13)

The equation for changing the amount of parallax d that makes thechanged depth Z′ be the same as the original depth Z is considered.

By changing the amount of parallax d in equation (13), equation (14) isobtained.Z′(d′)=L·d′/(S·E+d′)   (14)

Considering the equation d′=f(d) that makes the changed depth Z′ be thesame as the original depth Z, equations (15) are established fromequation (13) and equation (14).Z=Z′L·d/(E+d)=L·d′/(S·E+d′)d′·(E+d)=d·(S·E+d′)d′·(E+d′·d)=d·S·E+d·d′d′=S·d   (15)

Accordingly, when the inter-eye distance is changed from the originalinter-eye distance by a factor S, the amount of parallax is changed by afactor S, thereby canceling a change in the depth caused by a change inthe inter-eye distance.

In the foregoing description, the viewing distance, the display size,and the inter-eye distance before being changed correspond to intendedviewing environment parameters, while the viewing distance, the displaysize, and the inter-eye distance after being changed correspond toactual viewing environment parameters.

FIG. 18 illustrates a summary of processing for canceling a change inthe depth, which is caused by the difference between intended viewingenvironment parameters and actual viewing environment parameters, suchas the viewing distance, the display size, and the inter-eye distance.

In FIG. 18, the factor S that changes the viewing environment parametersis set to be 4 or ¼, i.e., the viewing distance, the display size, orthe inter-eye distance is increased to four times or reduced to ¼ of theoriginal viewing environment parameter.

In FIG. 18, when the amount of correction is a combination of thevirtual inter-camera distance and the image scaling amount, the controlratio of the virtual inter-camera distance to the image scaling amountis 50:50.

In images obtained by controlling the virtual inter-camera distance,artifacts occur if the images have occlusion. In images obtained bycontrolling the image scaling amount, the aspect ratio is changed. Thus,by a combination of the virtual inter-camera distance control and theimage scaling control, a change in the aspect ratio can be suppressedwhile reducing the occurrence of artifacts. In this case, when the imagescaling amount and the virtual inter-camera distance are designated byRs and Rc, respectively, and when the magnification of the necessaryamount of parallax is represented by R′, Rs and Rc that satisfy R′=Rs·Rccan be used.

A description is given in detail below, with reference to the flowchartof FIG. 19, of correction processing performed by the correctionprocessor 54 having the first configuration described above, i.e., firstcorrection processing in step S34 in FIG. 8.

In this embodiment, it is assumed that the control ratio of the virtualinter-camera distance to the image scaling amount is 50:50. Also forsimple description, it is assumed that only one of the three viewingenvironment parameters, such as the viewing distance, the display size,and the inter-eye distance, is different between intended viewingenvironments and actual viewing environments.

In step S51, the parallax control parameter calculator 81 obtainsintended viewing environment parameters supplied from the reader 51 andactual viewing environment parameters supplied from the operation inputunit 55. The actual viewing environment parameters input into theoperation input unit 55 may be stored in a predetermined storage unit inadvance. In this case, the parallax control parameter calculator 81 mayobtain the actual viewing environment parameters from the storage unit.

In step S52, the parallax detector 91 detects the amount of parallaxbetween each pixel of a left-eye image and the corresponding pixel of aright-eye image supplied from the video decoder 53 according to, forexample, a block matching method. The detected amounts of parallax ofthe corresponding pixels between the left-eye image and the right-eyeimage are supplied to the virtual inter-camera distance image generator92 as, for example, a parallax map.

In step S53, the parallax control parameter calculator 81 compares thesupplied intended viewing environment parameters with actual viewingenvironment parameters so as to determine whether the viewing distanceis different. If it is determined in step S53 that the viewing distanceis different, the process proceeds to step S54. In step S54, theparallax control parameter calculator 81 determines the virtualinter-camera distance in accordance with a change in the viewingdistance. The virtual inter-camera distance is determined from equations(11) and equation (12). The viewing distance before being changed isthat of the intended viewing environment parameter, while the viewingdistance after being changed is that of the actual viewing environmentparameter. The parallax control parameter calculator 81 supplies thedetermined virtual inter-camera distance to the virtual inter-cameradistance image generator 92 as the parallax control parameter.

In step S55, the virtual inter-camera distance image generator 92generates a left-eye image and a right-eye image captured with thevirtual inter-camera distance supplied from the parallax controlparameter calculator 81, and supplies the generated left-eye image andright-eye image to the image scaling unit 93. The image scaling unit 93outputs the supplied left-eye image and right-eye image without changingthem.

If it is determined in step S53 that the viewing distance is the same,steps S54 and S55 are skipped, and the process proceeds to step S56.

In step S56, the parallax control parameter calculator 81 compares thesupplied intended viewing environment parameters with the suppliedactual viewing environment parameters so as to determine whether thedisplay size is different. If it is determined in step S56 that thedisplay size is different, the process proceeds to step S57. In stepS57, the parallax control parameter calculator 81 determines the virtualinter-camera distance and the image scaling amount in accordance with achange in the display size. Since the control ratio of the virtualinter-camera distance to the image scaling amount is 50:50, the parallaxcontrol parameter calculator 81 determines each of the virtualinter-camera distance and the image scaling amount used for correctinghalf the amount of change of parallax in accordance with a change in thedisplay size. The parallax control parameter calculator 81 supplies thedetermined virtual inter-camera distance and the determined imagescaling amount to the virtual inter-camera distance image generator 92and the image scaling unit 93, respectively.

Then, in step S58, the virtual inter-camera distance image generator 92generates a left-eye image and a right-eye image captured with thevirtual inter-camera distance supplied from the parallax controlparameter calculator 81, and supplies the generated left-eye image andright-eye image to the image scaling unit 93.

In step S59, the image scaling unit 93 generates a left-eye image and aright-eye image with the image scaling amount supplied from the parallaxcontrol parameter calculator 81, and outputs the generated left-eyeimage and right-eye image.

If it is determined in step S56 that the display size is the same, stepsS57 through S59 are skipped, and the process proceeds to step S60.

In step S60, the parallax control parameter calculator 81 compares thesupplied intended viewing environment parameters with the actual viewingenvironment parameters so as to determine whether the inter-eye distanceis different. If it is determined in step S60 that the inter-eyedistance is different, the process proceeds to step S61. In step S61,the parallax control parameter calculator 81 determines the virtualinter-camera distance and the image scaling amount in accordance with achange in the inter-eye distance. Since the control ratio of the virtualinter-camera distance to the image scaling amount is 50:50, the parallaxcontrol parameter calculator 81 determines each of the virtualinter-camera distance and the image scaling amount used for correctinghalf the amount of change of parallax in accordance with a change in theinter-eye distance. The parallax control parameter calculator 81supplies the determined virtual inter-camera distance and the determinedimage scaling amount to the virtual inter-camera distance imagegenerator 92 and the image scaling unit 93, respectively.

Then, in step S62, the virtual inter-camera distance image generator 92generates a left-eye image and a right-eye image captured with thevirtual inter-camera distance supplied from the parallax controlparameter calculator 81, and supplies the generated left-eye image andright-eye image to the image scaling unit 93.

In step S63, the image scaling unit 93 generates a left-eye image and aright-eye image with the image scaling amount supplied from the parallaxcontrol parameter calculator 81, and outputs the generated left-eyeimage and right-eye image. The processing is then completed.

As described above, with the correction processor 54 having the firstconfiguration, by changing the virtual inter-camera distance or/and theimage scaling amount, three-dimensional distortions perceived by theuser, which are caused by the difference between intended viewingenvironment parameters and actual viewing environment parameters, can becorrected.

In the example described above, the first correction processing isperformed when only one of the three viewing environment parameters,such as the viewing distance, the display size, and the inter-eyedistance, is different. However, when two or more viewing environmentparameters are different, processing may also be performed in a mannersimilar to that discussed above. For example, if the viewing distanceand the display size are different, the product of the virtualinter-camera distance changed due to a change in the viewing distanceand the virtual inter-camera distance changed due to a change in thedisplay size is the overall virtual inter-camera distance, which servesas the amount of correction.

In the example described above, the control ratio of the virtualinter-camera distance to the image scaling amount is 50:50. However, ifcorrection is made only by using one of the virtual inter-cameradistance and the image scaling amount, steps S58 and S62 or steps S59and S63 are omitted.

4. Second Example Of Correction Processor Of Playback Apparatus

Correction processing performed by the correction processor 54 having asecond configuration is discussed below.

In the above-described correction processor 54 having the firstconfiguration, three-dimensional distortions are corrected by changingthe virtual inter-camera distance and the image scaling amount. Incontrast, in the correction processor 54 having the secondconfiguration, three-dimensional distortions caused by the differencebetween intended viewing environment parameters and actual viewingenvironment parameters are corrected by shifting a left-eye image and aright-eye image in a direction in which parallax occurs (in thehorizontal direction). In this case, the parallax control parameter isthe amount by which image pixels forming an image are shifted.

FIG. 20 illustrates an example of left-eye images and right-eye imagesbefore and after corrections are made by shifting the images.

In FIG. 20, the left-eye image and the right-eye image shown as theoriginal image are images before being corrected.

In FIG. 20, the left-eye image and the right-eye image shown as theoutward shifted image are obtained by shifting the original left-eyeimage and right-eye image outward by a total of 100 pixels (−100 pixels)so that the depth is changed in the backward direction (away from theviewer).

In FIG. 20, the left-eye image and the right-eye image shown as theinward shifted image are obtained by shifting the original left-eyeimage and right-eye image inward by a total of 100 pixels (+100 pixels)so that the depth is changed in the forward direction (toward theviewer).

FIG. 21 illustrates a change in the amount of parallax when the image isshifted in the outward direction and in the inward direction, as shownin FIG. 20.

As shown in FIG. 21, when the image is shifted outward, the depth ischanged in the backward direction, and when the image is shifted inward,the depth is changed in the forward direction.

In correction processing utilizing image shifting, it is difficult tocompletely correct for three-dimensional distortions perceived by auser, which are caused by the difference between intended viewingenvironment parameters and actual viewing environment parameters. Thus,in the correction processor 54 having the second configuration, precisecorrections are made to a predetermined point of the amounts ofparallax, such as the average, the maximum, the minimum, or the mode, ofthe amounts of parallax between the pixels of the left-eye image and theassociated pixels of the right-eye image.

For example, FIG. 22A illustrates the frequency distribution of theamounts of parallax between the pixels of a left-eye image and theassociated pixels of a right-eye image. If the display size of actualviewing environments is twice as large as that of intended viewingenvironments, the frequency distribution of the amounts of parallaxbetween the left-eye image and the right-eye image viewed by the user isshown in FIG. 22B.

In the frequency distribution of the amounts of parallax in the actualviewing environments shown in FIG. 22B, since the display size isdoubled, the depth also becomes doubled. More specifically, the depthbackward from the display unit is changed further in the backwarddirection, and the depth forward from the display unit is changedfurther in the forward direction. The amount of parallax becomes twiceas large as that in the intended viewing environments with respect tothe parallax 0.

FIG. 22C shows the frequency distribution obtained by shifting thefrequency distribution of the amounts of parallax in the actual viewingenvironments indicated by FIG. 22B so that a predetermined point of theamount of parallax, for example, the mode of the amount of parallax,shown in FIG. 22B coincides with that of FIG. 22A.

In this manner, in the correction processor 54 having the secondconfiguration, image shifting processing for precisely correcting apredetermined point of the amounts of parallax between a left-eye imageand a right-eye image is performed.

FIG. 23 is a block diagram illustrating an example of the secondconfiguration of the correction processor 54 that performs theabove-described image shifting processing.

The correction processor 54 shown in FIG. 23 includes a parallax controlparameter calculator 101 and a parallax controller 102. The parallaxcontroller 102 includes a parallax detector 111 and an image shiftingunit 112.

Intended viewing environment parameters received from the reader 51 andactual viewing environment parameters received from the operation inputunit 55 are supplied to the parallax control parameter calculator 101. Aparallax map is also supplied to the parallax control parametercalculator 101 from the parallax detector 111 of the parallax controller102.

The parallax control parameter calculator 101 generates, based on theparallax map supplied from the parallax detector 111, a frequencydistribution of the amounts of parallax between the pixels of a left-eyeimage and the associated pixels of a right-eye image.

The parallax control parameter calculator 101 calculates the amount bywhich an image is to be shifted (such an amount is hereinafter referredto as the “image shifting amount”). This image shifting is performed byshifting a predetermined reference point of the generated frequencydistribution on the basis of the difference between intended viewingenvironment parameters and actual viewing environment parameters. Inresponse to an instruction given from a user utilizing, for example, asetting screen, the parallax control parameter calculator 101 storestherein which one of the average, the maximum, the minimum, and the modeof the amounts of parallax between the pixels of a left-eye image andthe associated pixels of a right-eye image is used as the referencepoint. The parallax control parameter calculator 101 supplies thecalculated image shifting amount to the image shifting unit 112 of theparallax controller 102.

As in the parallax detector 91 of the correction processor 54 having thefirst configuration shown in FIG. 9, the parallax detector 111 detectsthe amount of parallax between each pixel of a left-eye image and theassociated pixel of a right-eye image by using, for example, a blockmatching method. The resulting parallax map is supplied to the parallaxcontrol parameter calculator 101.

The image shifting unit 112 executes an image shifting operation on theleft-eye image and the right-eye image, which form the original image(before being corrected) supplied from the video decoder 53 (FIG. 7) inaccordance with the image shifting amount supplied from the parallaxcontrol parameter calculator 101. Then, the image data of the left-eyeimage and the right-eye image subjected to the image shifting operationis output to the display unit 61 (FIG. 7).

A description is now given of the relationship between a change in theimage shifting amount and a change in each of the viewing environmentparameters, such as the viewing distance, the inter-eye distance, andthe display size.

The image shifting amount when the original viewing distance L ischanged by a factor S, i.e., L′=S·L, is discussed first.

The depth Za when the original viewing distance L is changed to theviewing distance L′ by a factor S, i.e., L′=S·L, is expressed byequation (2). Assuming that the depth Za is indicated by Z′, equation(2) can be expressed by equation (8).

The equation for correcting the amount of parallax d that makes thechanged depth Z′ be the same as the original depth Z is considered.Assuming that the corrected amount of parallax is designated by d′ andthe image shifting amount to be determined is indicated by “shift”,equation (16) is defined as follows.d′=f(d)=d+shift   (16)

Considering the equation d′=f(d) that makes the changed depth Z′ be thesame as the original depth Z, the image shifting amount can bedesignated by equations (17) by using equations (11).d′=d·E/(S·E+S·d−d)d+shift=d·E/(S·E+S·d−d)shift=d·E/(S·E+S·d−d)−d   (17)

Accordingly, when the original viewing distance is changed by a factorS, in order to cancel such a change in the viewing distance, an image isshifted by the image shifting amount expressed by equations (17).

For example, when the viewing distance L is 1500 mm, the display size is46 inches, the inter-eye distance E is 65 mm, and the maximum amount ofparallax is 20 pixels (pixel pitch is 0.53 mm per pixel and the lengthsof the pixels are 10.6 mm), the image shifting amount “shift” when theviewing distance is increased twice is −11 pixels as calculated in thefollowing manner.shift=10.6×65/(2.0×65+2.0×10.6−10.6)−10.6=−5.7 mm=−10.8 pixels≈−11 pixels

The image shifting amount when the original inter-eye distance ischanged by a factor S is discussed below.

In order to cancel a change in the inter-eye distance by a factor S, itis necessary to change the amount of parallax by a factor S, asindicated by equations (15). Since d′=d+shift as expressed by equation(16), the image shifting amount “shift” can be expressed by equations(18).d′=S·dd+shift=S·dshift=(S−1)·d   (18)

Accordingly, when the original inter-eye distance is changed by a factorS, in order to cancel such a change in the inter-eye distance, an imageis shifted by the image shifting amount “shift” expressed by equations(18). The amount of parallax d in equations (18) indicates the amount ofparallax before shifting the reference point, such as the average, themaximum, the minimum, or the mode, of the amounts of parallax betweenthe pixels of a left-eye image and the associated pixels of a right-eyeimage.

For example, when the maximum of the amounts of parallax, which is thereference point before being shifted, is 20 pixels, and when theinter-eye distance is increased to be twice as large as the originalinter-eye distance, the image shifting amount “shift” is calculated tobe 20 pixels as follows.shift=(2−1)×20=20

FIG. 24 illustrates image shifting correction processing for canceling achange in the depth, which is caused by the difference between intendedviewing environment parameters and actual viewing environmentparameters, such as the viewing distance, the display size, and theinter-eye distance.

In FIG. 24, the factor S that changes the viewing environment parametersis set to be 4 or ¼, i.e., the viewing distance, the display size, orthe inter-eye distance is increased to four times or reduced to ¼ of theoriginal viewing environment parameter.

A description is given in detail below, with reference to the flowchartof FIG. 25, of second correction processing performed by the correctionprocessor 54 having the second configuration described above, i.e.,second correction processing in step S34 in FIG. 8. As in the processingshown in FIG. 19, it is assumed that only one of the three viewingenvironment parameters, such as the viewing distance, the display size,and the inter-eye distance, is different between intended viewingenvironments and actual viewing environments.

In step S71, the parallax control parameter calculator 101 obtainsintended viewing environment parameters supplied from the reader 51 andactual viewing environment parameters supplied from the operation inputunit 55.

In step S72, the parallax detector 111 detects the amount of parallaxbetween each pixel of a left-eye image and the associated pixel of aright-eye pixel supplied from the video decoder 53 according to, forexample, a block matching method. The detected amounts of parallax ofthe corresponding pixels between the left-eye image and the right-eyeimage are supplied to the parallax control parameter calculator 101 as,for example, a parallax map.

In step S73, the parallax control parameter calculator 101 determinesthe image shifting amount in accordance with a change in the viewingdistance, the display size, or the inter-eye distance. The determinedimage shifting amount is supplied to the image shifting unit 112.

In step S74, the image shifting unit 112 generates a left-eye image anda right-eye image shifted by the image shifting amount supplied from theparallax control parameter calculator 101. The image shifting unit 112then outputs image data of the generated left-eye image and right-eyeimage. The processing is then completed.

As described above, with the correction processor 54 having the secondconfiguration, by performing image shifting processing,three-dimensional distortions perceived by the user, which are caused bythe difference between intended viewing environment parameters andactual viewing environment parameters, can be corrected.

In the above-described example, the correction processor 54 has one ofthe first configuration and the second configuration. However, thecorrection processor 54 may have both the first configuration and thesecond configuration so as to perform the correction processing by usingany of the first configuration and the second configuration. In thiscase, a user may select which of the first correction processing or thesecond correction processing is to be performed, or the processing maybe selected according to the type of picture to be played back.

Additionally, three types of viewing environment parameters, such as theviewing distance, the display size, and the inter-eye distance, areused. However, only one or two of these parameters may be used.

In the above-described example, an embodiment of the present inventionis applied to a playback apparatus. However, it may be applied toapparatuses other than a playback apparatus. For example, 3D contentvideo data and intended viewing environment parameters may be providedby transmission via a network, such as satellite broadcasting, cabletelevision (TV), or the internet. Accordingly, an embodiment of thepresent invention may be applicable to a display apparatus or arecording/playback apparatus that receives 3D content video data andintended viewing environment parameters sent by transmission via anetwork, corrects 3D pictures in accordance with the difference betweenthe intended viewing environment parameters and actual viewingenvironment parameters, and displays the resulting pictures. Anembodiment of the present invention may be used as an image processingapparatus that obtains actual viewing environment parameters as well as3D content video data and intended viewing environment parameters, andthen corrects 3D video data and outputs corrected 3D video data.

5. Computer to which an Embodiment of the Present Invention Is Applied

The above-described series of processing operations may be executed byhardware or software. If software is used, a program forming thatsoftware is installed in, for example, a general-purpose computer.

FIG. 26 illustrates an example of the configuration of a computer inwhich a program executing the above-described series of processingoperations is to be installed according to an embodiment of the presentinvention.

The program may be recorded in advance on a storage unit 208 or a readonly memory (ROM) 202, which serves as a recording medium built in thecomputer.

Alternatively, the program may be stored (recorded) in removable media211. The removable media 211 may be provided as so-called packagesoftware. The removable media 211 may include a compact disc read onlymemory (CD-ROM), a magneto-optical (MO) disc, a digital versatile disk(DVD), a magnetic disk, a semiconductor memory, etc.

The program may be installed into the computer from the above-describedremovable media 211 via a drive 210. Alternatively, the program may bedownloaded into the computer via a communication network or abroadcasting network and be installed in the built-in storage unit 208.That is, the program may be received by a communication unit 209 via acabled or wireless transmission medium and be installed in the storageunit 208.

The computer has a built-in central processing unit (CPU) 201, and aninput/output interface 205 is connected to the CPU 201 via a bus 204.

The CPU 201 executes the program stored in the ROM 202 in response to aninstruction input from a user by operating an input unit 206 via theinput/output interface 205. Alternatively, the CPU 201 may load theprogram stored in the storage unit 208 into a random access memory (RAM)203 and execute the loaded program.

The CPU 201 executes processing in accordance with the above-describedflowcharts or operations performed by the elements of theabove-described apparatuses. Then, if necessary, the CPU 201 outputs theprocessing results from an output unit 207, or transmits them from thecommunication unit 209, or records them in the storage unit 208 via theinput/output interface 205.

The input unit 206 may include a keyboard, a mouse, a microphone, etc.The output unit 207 may include a liquid crystal display (LCD), aspeaker, etc.

In this specification, it is not necessary that the processing executedby the computer in accordance with the program be performed inchronological order, as in the order shown in the flowcharts. That is,the processing executed by the computer in accordance with the programincludes processing executed in parallel or individually (e.g. parallelprocessing or object processing).

The program may be executed by a single computer (processor) or may beexecuted by a plurality of computers (distribute processing). Theprogram may be transferred to a remote computer and be executed.

In the above-described embodiment, two-view 3D pictures having twoviewpoints have been discussed. However, multi-view 3D pictures havingthree or more numbers of viewpoints may also be applied to an embodimentof the present invention.

Embodiments of the present invention are not restricted to theabove-described embodiments, but various modifications may be madewithout departing from the gist of the invention.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2010-092816 filedin the Japan Patent Office on Apr. 14, 2010, the entire contents ofwhich are hereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing apparatus comprising: a firstacquiring circuitry configured to acquire an intended viewingenvironment parameter, which is a parameter of an intended viewingenvironment, together with image data of an original three-dimensionalpicture; a second acquiring circuitry configured to acquire an actualviewing environment parameter, which is a parameter of an actual viewingenvironment for a user viewing the original three-dimensional picture,wherein each of the intended viewing environment parameter and theactual viewing environment parameter includes an inter-eye distance; anda correction processing circuitry configured to correct the originalthree-dimensional picture in accordance with a difference between theacquired intended viewing environment parameter and the acquired actualviewing environment parameter, wherein the correction processingcircuitry is also configured to: determine a control ratio between avirtual inter-camera distance and an image scaling amount; generate afirst corrected three-dimensional picture based on the originalthree-dimensional picture, the first corrected three-dimensional pictureincluding a virtual inter-camera distance determined based on thecontrol ratio; and perform image scaling on the first correctedthree-dimensional picture based on the image scaling amount determinedfrom the control ratio, the image scaling generating a second correctedthree-dimensional picture.
 2. The image processing apparatus accordingto claim 1, wherein each of the intended viewing environment parameterand the actual viewing environment parameter includes at least one of aviewing distance, and a display size.
 3. The image processing apparatusaccording to claim 2, wherein: the acquired image data of the originalthree-dimensional picture includes a left-eye image and a right-eyeimage captured with a predetermined inter-camera distance; and thecorrection processing circuitry includes a virtual inter-camera distanceimage generating circuitry configured to generate a left-eye image and aright-eye image captured with the virtual inter-camera distance changedfrom the predetermined inter-camera distance, an image scalingconfigured to scale the left-eye image and the right-eye image generatedby the virtual inter-camera distance image generating circuitry, and acontrol parameter calculating circuitry configured to calculate thevirtual inter-camera distance used in the virtual inter-camera distanceimage generating circuitry, or the image scaling amount used in theimage scaling circuitry, from a difference between the inter-eyedistance, the viewing distance, or the display size forming the intendedviewing environment parameter, and the inter-eye distance, the viewingdistance, or the display size forming the actual viewing environmentparameter.
 4. The image processing apparatus according to claim 2,wherein: the acquired image data of the original three-dimensionalpicture includes a left-eye image and a right-eye image; and thecorrection processing circuitry includes image shifting processingcircuitry for shifting the left-eye image and the right-eye image sothat an amount of parallax between the left-eye image and the right-eyeimage becomes a predetermined amount, and control parameter calculatingcircuitry for calculating an image shifting amount, so that the amountof parallax between the left-eye image and the right-eye image becomesthe predetermined amount, from a difference between the inter-eyedistance, the viewing distance, or the display size forming the intendedviewing environment parameter, and the inter-eye distance, the viewingdistance, or the display size forming the actual viewing environmentparameter.
 5. An image processing method for an image processingapparatus including a first acquiring circuitry configured to acquiredata, a second acquiring circuitry configured to acquire data, and acorrection processing circuitry configured to correct an originalthree-dimensional picture, the image processing method comprising thesteps of: acquiring by the first acquiring circuitry an intended viewingenvironment parameter, which is a parameter of an intended viewingenvironment, together with image data of the original three-dimensionalpicture; acquiring by the second acquiring circuitry an actual viewingenvironment parameter, which is a parameter of an actual viewingenvironment for a user viewing the original three-dimensional picture,wherein each of the intended viewing environment parameter and theactual viewing environment parameter includes an inter-eye distance; andcorrecting by the correction processing circuitry the originalthree-dimensional picture in accordance with a difference between theacquired intended viewing environment parameter and the acquired actualviewing environment parameter, wherein the correction processingcircuitry is also configured to: determine a control ratio between avirtual inter-camera distance and an image scaling amount; generate afirst corrected three-dimensional picture based on the originalthree-dimensional picture, the first corrected three-dimensional pictureincluding a virtual inter-camera distance determined based on thecontrol ratio; and perform image scaling on the first correctedthree-dimensional picture based on the image scaling amount determinedfrom the control ratio, the image scaling generating a second correctedthree-dimensional picture.
 6. A non-transitory computer-readable mediumhaving stored therein a program that comprises instructions for allowinga computer to execute a method comprising the steps of: acquiring anintended viewing environment parameter, which is a parameter of anintended viewing environment, together with image data of an originalthree-dimensional picture; acquiring an actual viewing environmentparameter, which is a parameter of an actual viewing environment for auser viewing the original three-dimensional picture, wherein each of theintended viewing environment parameter and the actual viewingenvironment parameter includes an inter-eye distance; and correcting theoriginal three-dimensional picture in accordance with a differencebetween the acquired intended viewing environment parameter and theacquired actual viewing environment parameter, wherein correcting theoriginal three-dimensional picture includes: determining a control ratiobetween a virtual inter-camera distance and an image scaling amount;generating a first corrected three-dimensional picture based on theoriginal three-dimensional picture, the first correctedthree-dimensional picture including a virtual inter-camera distancedetermined based on the control ratio; and performing image scaling onthe first corrected three-dimensional picture based on the image scalingamount determined from the control ratio, the image scaling generating asecond corrected three-dimensional picture.
 7. An image processingapparatus comprising: a first acquiring unit configured to acquire anintended viewing environment parameter, which is a parameter of anintended viewing environment, together with image data of an originalthree-dimensional picture; a second acquiring unit configured to acquirean actual viewing environment parameter, which is a parameter of anactual viewing environment for a user viewing the originalthree-dimensional picture, wherein each of the intended viewingenvironment parameter and the actual viewing environment parameterincludes an inter-eye distance; and a correction processor configured tocorrect the original three-dimensional picture in accordance with adifference between the acquired intended viewing environment parameterand the acquired actual viewing environment parameter, wherein thecorrection processor is configured to determine a control ratio betweena virtual inter-camera distance and an image scaling amount; generate afirst corrected three-dimensional picture based on the originalthree-dimensional picture, the first corrected three-dimensional pictureincluding a virtual inter-camera distance determined based on thecontrol ratio; and perform image scaling on the first correctedthree-dimensional picture based on the image scaling amount determinedfrom the control ratio, the image scaling generating a second correctedthree-dimensional picture.
 8. The image processing method of claim 5,wherein each of the intended viewing environment parameter and theactual viewing environment parameter includes at least one of a viewingdistance, and a display size.
 9. The image processing method of claim 5,wherein the acquired image data of the original three-dimensionalpicture includes a left-eye image and a right-eye image captured with apredetermined inter-camera distance, and wherein the method furthercomprises: generating a left-eye image and a right-eye image capturedwith the virtual inter-camera distance changed from the predeterminedinter-camera distance; scaling the left-eye image and the right-eyeimage; and calculating the virtual inter-camera distance, or the imagescaling amount, from a difference between the inter-eye distance, theviewing distance, or the display size forming the intended viewingenvironment parameter, and the inter-eye distance, the viewing distance,or the display size forming the actual viewing environment parameter.10. The method of claim 8, wherein the acquired image data of theoriginal three-dimensional picture includes a left-eye image and aright-eye image, and wherein the method further comprises: shifting theleft-eye image and the right-eye image so that an amount of parallaxbetween the left-eye image and the right-eye image becomes apredetermined amount; and calculating an image shifting amount, so thatthe amount of parallax between the left-eye image and the right-eyeimage becomes the predetermined amount, from a difference between theinter-eye distance, the viewing distance, or the display size formingthe intended viewing environment parameter, and the inter-eye distance,the viewing distance, or the display size forming the actual viewingenvironment parameter.
 11. The non-transitory computer-readable mediumof claim 6, wherein each of the intended viewing environment parameterand the actual viewing environment parameter includes at least one of aviewing distance, and a display size.
 12. The non-transitorycomputer-readable medium of claim 11, wherein the acquired image data ofthe original three-dimensional picture includes a left-eye image and aright-eye image captured with a predetermined inter-camera distance, andwherein the method further comprises: generating a left-eye image and aright-eye image captured with the virtual inter-camera distance changedfrom the predetermined inter-camera distance; scaling the left-eye imageand the right-eye image; and calculating the virtual inter-cameradistance, or the image scaling amount, from a difference between theinter-eye distance, the viewing distance, or the display size formingthe intended viewing environment parameter, and the inter-eye distance,the viewing distance, or the display size forming the actual viewingenvironment parameter.
 13. The non-transitory computer-readable mediumof claim 11, wherein the acquired image data of the originalthree-dimensional picture includes a left-eye image and a right-eyeimage, and wherein the method further comprises: shifting the left-eyeimage and the right-eye image so that an amount of parallax between theleft-eye image and the right-eye image becomes a predetermined amount;and calculating an image shifting amount, so that the amount of parallaxbetween the left-eye image and the right-eye image becomes thepredetermined amount, from a difference between the inter-eye distance,the viewing distance, or the display size forming the intended viewingenvironment parameter, and the inter-eye distance, the viewing distance,or the display size forming the actual viewing environment parameter.14. The image processing apparatus of claim 7, wherein each of theintended viewing environment parameter and the actual viewingenvironment parameter includes at least one of a viewing distance, and adisplay size.
 15. The image processing apparatus of claim 14, whereinthe acquired image data of the original three-dimensional pictureincludes a left-eye image and a right-eye image captured with apredetermined inter-camera distance; and wherein the correctionprocessor includes a virtual inter-camera distance image generating unitfor generating a left-eye image and a right-eye image captured with thevirtual inter-camera distance changed from the predeterminedinter-camera distance, an image scaling unit for scaling the left-eyeimage and the right-eye image generated by the virtual inter-cameradistance image generating unit, and a control parameter calculating unitfor calculating the virtual inter-camera distance used in the virtualinter-camera distance image generating unit, or the image scaling amountused in the image scaling unit, from a difference between the inter-eyedistance, the viewing distance, or the display size forming the intendedviewing environment parameter, and the inter-eye distance, the viewingdistance, or the display size forming the actual viewing environmentparameter.
 16. The image processing apparatus of claim 14, wherein: theacquired image data of the original three-dimensional picture includes aleft-eye image and a right-eye image; and the correction processorincludes an image shifting processing unit for shifting the left-eyeimage and the right-eye image so that an amount of parallax between theleft-eye image and the right-eye image becomes a predetermined amount,and a control parameter calculating unit for calculating an imageshifting amount, so that the amount of parallax between the left-eyeimage and the right-eye image becomes the predetermined amount, from adifference between the inter-eye distance, the viewing distance, or thedisplay size forming the intended viewing environment parameter, and theinter-eye distance, the viewing distance, or the display size formingthe actual viewing environment parameter.